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ABSTRACT 

Image  flow,  the  apparent  motion  of  brightness  patterns  on  the  image  plane,  can  provide  important  visual  informa- 
tion such  as  distance,  shape,  surface  orientation,  and  boundaries.  It  can  be  determined  by  either  feature  tracking  or 
spatio-temporal  analysis.  We  consider  spatio-temporal  methods,  and  show  how  differential  range  can  be  estimated  from 
time-space  imagery. 

We  generate  a time-space  image  by  considering  only  one  scan  line  of  the  image  obtained  from  a camera  moving 
in  the  horizontal  direction  at  each  lime  interval.  At  the  next  instant  of  time,  we  shift  the  previous  line  up  by  one  pixel, 
and  obtain  another  line  from  the  image.  We  continue  the  procedure  to  obtain  a time-space  image,  where  each  horizon- 
tal line  represents  the  spatial  relationship  of  the  pixels,  and  each  vertical  line  the  temporal  relationship. 

Each  feature  along  the  horizontal  scan  line  generates  an  edge  in  the  time-space  image,  the  slope  of  which 
depends  upon  the  distance  of  the  feature  from  the  camera.  We  apply  two  mutually  perpendicular  edge  operators  to  the 
time-space  image,  and  determine  the  slope  of  each  edge.  We  show  that  this  corresponds  to  optical  flow.  We  use  the 
result  to  obtain  the  differential  range,  and  show  how  this  can  be  implemented  on  the  Pipelined  Image  Processing 
Engine  (PIPE). 

We  use  a simple  technique  to  calibrate  the  camera  and  show  how  the  depth  can  be  obtained  from  optical  flow. 
We  provide  a statistical  analysis  of  the  results  of  3-D  reconsuuction  of  the  scenes  using  optical  flow  determined  from 
3x3,  5x5,  and  7x7  edge  operators. 


1.  INTRODUCTION 

Image  understanding  is  concerned  with  the  determination  of  visual  information  such  as  distance,  shape,  surface 
orientation,  boundaries,  etc.  One  method  of  determining  the  visual  information  using  monocular  vision  is  to  use  optical 
flow,  which  is  defined  as  the  apparent  motion  of  brightness  patterns  in  the  image  plane.  There  are  two  methods  to  find 
the  optical  flow. 

1.  Feature  matching  based  method.  In  this  method,  identifiable  features  from  a sequence  of  images  are  extracted 
and  correspondence  is  established.  The  corresponding  features  are  used  to  calculate  a set  of  disparity  vectors  for 
the  sequence.  Any  identifiable  entity  can  be  used  as  a feature,  but  sharp,  localized  features  give  the  best  accu- 
racy. 

2.  Spatio-iemjxtral  methods.  These  methods  use  the  spatial  and  temjxaral  relationships  of  image  intensities  to 
determine  the  optical  flow.  The  time  history  of  the  position  of  a feature  in  the  image  can  be  represented  in  a 3- 
dimensional  coordinate  system  (x,y,f).  This  is  often  called  a spatio-tempwral  solid  or  spatio-tempwral  volume. 
The  spatio-tempxDral  area  is  defined  as  an  x-f  slice  of  the  solid,  i.e.,  a slice  taken  with  y = constant.  The  slope 
at  any  px)int  (x,0  in  the  spatio-iempxjral  area  gives  the  horizontal  component  of  its  velocity.  Therefore,  the  hor- 
izontal component  of  optical  flow  can  be  obtained  by  determining  the  orientation  of  the  edges. 

Our  papor  is  organized  as  follows. 
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In  section  2,  we  define  optical  flow,  and  show  the  equations  for  estimating  optical  flow  using  gradient  based 
methods.  We  also  show  how  to  reconstruct  a scene  using  optical  flow.  In  section  3,  we  give  a brief  introduction  to  edge 
operators,  and  show  how  the  magnitude  and  direction  of  edges  can  be  extracted  using  edge  operators.  In  section  4,  we 
define  an  Epipolar  Plane  Image  (EPI),  and  show  how  distance  of  objects  from  the  camera  is  encoded  in  the  EPI.  Here, 
we  use  the  results  of  sections  2 and  3 to  show  how  optical  flow,  and  hence  the  depth  from  the  camera  can  be  extracted 
from  the  spatio-temporal  image.  The  algorithm  and  its  implementation  on  PIPE  are  given  in  section  5.  We  describe 
our  experimental  setup  and  the  experiments  in  section  6.  In  section  7,  we  provide  several  types  of  results  such  as 
the  differential  range,  and  the  number  of  misdassif ied  points  in  addition  to  the  mean  and  standard  deviation 
obtained  fi’om  several  types  of  edge  operators. 


2,  MOTION  FIELD  AND  OPTICAL  FLOW 

When  the  image  of  an  object  is  formed  on  the  image  plane  of  the  camera,  every  point  Pg  on  the  object  results  in 
a corresponding  point  P,  in  the  image.  If  Pg  moves  with  a velocity  Vg  in  the  real  world,  then  P^  moves  with  a velo- 
city Vj  in  the  image  plane.  Thus,  all  points  on  the  object  have  a corresponding  velocity  vector  in  the  image.  These 
velocity  vectors  constitute  the  motion  field.  As  the  object  moves,  the  brightness  pattern  in  the  image  plane  also  moves. 
The  apparent  motion  of  the  brighmess  pattern  is  known  as  optical  flow.  In  an  ideal  situation,  this  corresponds  to  the 
motion  field.^ 


Let  us  define  two  types  of  image  motion:  local  motion , which  is  due  to  the  motion  of  the  objects  in  the  environ- 
ment, and  global  motion , which  is  due  to  the  motion  of  the  camera  itself.  By  setting  the  local  motion  to  zero,  one  can 
obtain  an  equation  for  the  optical  flow  u induced  by  an  object  at  a depth  d in  terms  of  the  camera  velocity  V and  its 
focal  length  /. 


Using  similar  triangles  OAB  and  OPQ,  we  get: 


A = zK 
/ 


If  the  camera  is  moving  with  a uniform  velocity  V,  then  the 
numerator  of  the  above  equation  is  a constant.  The  equation  can  be 
written  as  follows. 


u 


(1) 


Q 


The  optical  flow  can  be  mathematically  modelled  and  used  as  an  estimate  of  the  motion  field,  but  this  requires 
certain  assumptions  which  are  listed  below. 

1.  The  apparent  velocity  of  brighmess  patterns  is  direcdy  identified  with  the  movement  of  the  surfaces  in  the  scene. 

2.  The  surface  being  imaged  is  flat,  and  thus  there  is  no  variation  in  brighmess  due  to  shading  effects. 

3.  Incidental  illumination  is  uniform  across  the  surface. 

Under  these  assumptions,  the  brighmess  at  a point  in  the  image  is  proportional  to  the  reflectance  of  the  surface  at 
the  corresponding  point  on  the  object  As  the  reflectance  varies  smoothly  and  has  no  discontinuities,  the  image  bright- 
ness is  also  continuous,  and  hence  differentiable.  This  property  of  the  image  intensity  is  used  by  gradient  based 
methods  to  determine  the  optical  flow.  In  a gradient  based  method,  the  image  intensity  function  I{x,yj)  in  the  image 
plane  at  a point  (x  ,y ) and  at  time  t is  expanded  using  Taylor’s  series  and  used  to  compute  the  optical  flow  field.^  The 
total  derivative  of  the  image  intensity  between  two  image  frames  separated  by  a very  small  time  interval  dt  is  zero. 

(2) 

dt 

It  should  be  noted  that  this  is  strictly  satisfied  only  for  translation  of  a rigid  body  in  planes  parallel  to  the  image  plane. 
Using  the  chain  rule  of  differentiation,  equation  (2)  can  be  written  as 

(3) 

dt  dx  dt  dy  dt  dt 
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This  can  be  written  as 


4u+/^v+/,=0  (4) 

where  u and  v are  the  components  of  flow  velocity  in  the  j:  and  y directions  respectively,  /,  and  ly  are  the  spatial  gra- 
dients, and  I,  is  the  temporal  gradient.  /, , ly  and  1,  can  be  measured  from  the  image  sequence,  and  the  optical  flow  (u , 
V ) can  be  computed.  Since  we  have  two  unknowns  u and  v , and  one  relationship  in  equation  (4),  the  problem  is  con- 
sidered ill-posed.  In  other  words,  we  need  one  more  constraint  to  regularize  the  problem  and  to  compute  the  flow  field. 
However,  if  the  direction  of  motion  of  the  camera  is  known,  the  optical  flow  can  be  computed  using  only  one  equation. 

Consider  a horizontally  moving  camera  where  the  motion  is  considered  to  be  along  the  x axis.  Since  the  com- 
ponent of  motion  in  the  y direction  is  zero,  equation  (4)  becomes 

/,u  +/,  =0 


The  optical  flow  in  the  x direction  is  given  by 


The  magnitude  of  optical  flow  is. 


(5) 


(6) 


The  use  of  this  method  is  justified  in  such  situations  where  the  direction  of  motion  of  the  camera  is  known.  Using 
Equations  (1)  and  (5),  an  equation  for  the  distance  d of  an  object  from  the  camera  can  be  obtained. 


d = 


E± 

I. 


(7) 


Thus,  knowing  K,  one  can  reconstruct  the  3D  geometry  of  a scene.  K can  be  determined  either  by  determining  / 

and  U,  or  by  using  the  EPI  as  shown  section  4.  In  this  work,  we  determine  /,  and  I,  using  several  edge  operators  on 

the  EPI,  and  reconstruct  the  3D  geometry  of  the  scene  using  Equation  (7).  We  also  provide  a statistical  analysis  of  the 
results  from  the  different  edge  operators. 

If  there  are  two  objects  0^  and  O2  at  depths  dj  and  J2.  respectively,  from  the  camera,  with  d]>d2,  and  they 
induce  optical  flow  values  Ui  and  U2,  then  we  can  derive  the  following  from  Equation  (1): 

d]-d2  U2-U] 

L (8) 

dx  U2 

We  define  Equation  (8)  as  the  differential  range.  The  reasons  why  we  calculate  this  function  are 

(i)  Equation  (8)  is  independent  of  V and  / so  that  calibration  of  the  camera  is  not  required,  and 

(ii)  we  would  like  to  determine  the  minimum  distance  between  two  objects  that  can  be  discriminated  using  optical 

flow,  and  express  the  result  as  a number  which  is  independent  of  the  depths  of  the  objects  from  the  camera. 


3.  EDGE  DETECTION 


An  edge  is  a geometric  form  whose  gray  level  is  consistent  on  each  of  the  two  adjacent,  extensive  regions  and 
changes  abruptly  as  the  border  between  the  regions  is  crossed.'^*  The  pixel  locations  where  this  abrupt  gray-level 
change  occurs  are  called  edge  points.  The  derivative  of  a continuous  image  lix,y)  has  a local  maximum  in  the  direc- 
tion perpendicular  to  the  edge.  The  magnitude  M and  the  direction  d>  are  given  by  the  following  equations.^ 


d)  = tan  ' -r 


(9) 

(10) 
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There  are  several  different  methods  by  which  edges  can  be  detected,  and  research  is  being  continued  to  determine  alter- 
native methods.  In  this  work,  which  is  an  application  of  edge  operators  to  estimate  optical  flow,  we  have  used  gradient 
operators.  The  gradient  operators  have  two  mutually  f)erpendicular  masks  //,  and  Hy  which  are  numerical  approxima- 
tions to  4 and  ly,  respectively,  and  are  used  to  determine  M and  <1>  using  equations  (9)  and  (10).  These  operators 
give  high  signal  to  noise  values  at  the  edge  points  and  zeroes  for  uniform  regions.  There  are  several  gradient  operators 
such  as  Prewitt,  Sobel,  isotropic,  stochastic,  etc.  AH  of  these  can  be  easily  implemented  on  digital  hardware. 

4.  SPATIO-TEMPORAL  IMAGE  ANALYSIS 

As  discussed  above,  the  time  history  of  each  feature  point  in  the  image  gives  the  spatio-temporal  volume,  and  a 
slice  in  the  temporal  direction  gives  the  spatio-temporal  area.  Figure  1(a)  shows  a stationary  scene  containing  a rail- 
road car  in  the  background  and  a tree  in  the  foreground.  Assume  that  the  camera  moves  in  the  horizontal  direction,  that 
the  optical  axis  of  the  camera  points  perpendicular  to  the  direction  of  motion,  and  that  the  optical  flow  is  in  the  nega- 
tive direction.  A horizontal  scan  line  j of  the  image  is  selected  and  the  spatio-temporal  area  for  this  scan  line  is 
obtained.  When  the  scan  line  is  coincident  with  the  direction  of  motion  of  the  camera,  the  spatio-temporal  area  is 
called  the  Epipolar  Plane  Image  (EPI).^  The  EPI  of  the  line  j is  shown  in  Figure  l{b).  The  slope  due  to  the  tree 
(closer  object)  is  smaller  than  the  slope  due  to  the  railroad  car  when  the  slope  is  measured  from  the  time  (vertical)  axis. 
Thus,  depending  upon  the  distance  of  the  objects  along  the  scan  line  j , there  will  be  lines  of  different  slopes  in  the 
EPI.  The  greater  the  depth  of  the  objects  from  the  camera,  the  greater  the  slope  measured  from  the  time  axis  will  be.  If 
the  slope  can  be  measured,  the  distance  between  the  camera  and  the  object  can  be  calculated. 


(ay  (^) 


Figure  1:  (a)  Image  at  time  k;  (b)  EPI  for  scan  line  j. 


The  use  of  time-space  imagery  for  determining  optical  flow  was  first  introduced  by  BoIIes  and  Baker.^  They 
determined  slope  by  using  a feature  matching  method.  Here  we  take  a different  approach  and  use  edge  operators  to 
determine  the  slope. 

In  section  2,  we  showed  how  optical  flow  can  be  obtained  using  the  ratio  of  temporal  difference  over  spatial 
difference  (Equation  6).  In  an  EPI,  the  horizontal  axis  represents  space  while  the  vertical  axis  represents  time.  Apply- 
ing //j  to  an  EPI  is  similar  to  finding  the  spatial  difference,  while  the  application  of  Hy  gives  the  temporal  difference. 
If  equation  (10)  is  modified  to  determine  the  slop)e  of  a line  in  an  EPI,  and  the  variable  y is  changed  to  / , it  can  be 
written  as  follows: 


tan(<l))  = 


(11) 
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Comparing  equations  (6)  and  (11),  one  can  see  that  the  two  represent  the  same  quantity.  We  use  Equation  (11)  to 
determine  optical  flow  in  the  EPI. 

A closer  look  at  Figure  (1)  and  Equation  (11)  shows  that  the  global  slope  of  an  object  in  the  EPI  is  equal  its 
optical  flow.  If  we  can  obtain  the  EPI  of  an  object  at  a known  distance  d from  the  camera  and  determine  the  global 
slope  manually,  we  can  determine  the  constant  K using  Equation  (1).  If  the  experiment  is  repeated  with  the  same  cam- 
era settings  and  the  same  value  for  V,  and  the  spatial  and  temporal  differences  are  determined  using  local  edge  opera- 
tors, we  can  reconstruct  the  scene  using  Equation  (7). 

In  section  3,  it  was  stated  that  the  gradient  operators  give  high  values  at  the  edge  points  and  zeroes  for  uniform 
regions.  Therefore,  optical  flow  will  have  high  values  at  the  edges  and  becomes  indeterminate  for  uniform  regions.  The 
signal-to-noise  of  the  flow  is  maximum  at  the  edge  and  decreases  as  one  moves  away  from  it  until  it  becomes  indeter- 
minate. To  find  a high  signal-to-noise  value  for  flow,  the  flow  value  at  the  edge  must  be  extracted,  while  the  flow 
values  around  the  edge  must  be  eliminated.  We  do  this  by  determining  the  zero  crossings  of  the  Laplacian-Gaussian  of 
the  image  and  returning  flow  only  at  these  zero  crossing  points. 

We  confine  ourselves  to  small  values  of  optical  flow  (less  than  2 pixels  per  frame-time)  because  flow  values 
significantly  greater  than  zero  are  known  to  induce  large  measurement  errors  due  to  under-sampling.^  We  have  imple- 
mented our  algorithm  to  compute  the  optical  flow  on  PIPE,  which  processes  images  at  video  rate. 

5.  PIPE  IMPLEMENTATION 

PIPE  is  a multi-stage,  multi-pipelined  image  processing  machine  which  may  be  used  as  the  front-end  of  a real- 
time image  understanding  system.  It  was  designed  specifically  for  low  level  vision  tasks  at  very  high  speed.  It  was  con- 
ceived and  designed  at  the  National  Institute  of  Standards  and  Technology  (formerly  National  Bureau  of  Standards)  by 
Kent,  et  al.*  and  is  commercially  available  through  Aspex  Incorporated.*  PIPE  has  a video  stage  to  receive  imagery,  an 
input  stage  with  four  buffers,  1 to  8 identical  Modular  Processing  Stages  (MPSs)  to  process  the  imagery,  and  an  output 
stage  with  four  buffers  to  store  images. 

For  further  details  on  PIPE,  refer  to  the  references  listed  in  this  paper."^"*  **-*^-*^ 

In  our  experiments,  we  have  used  PIPE  to  obtain  the  optical  flow  to  compute  the  differential  range  using  3x3, 
5x5,  and  7x7  edge  operators.  The  algorithm  to  determine  differential  range  of  a selected  scanline  of  an  image  has  the 
following  steps. 

1 . Obtain  the  EPI. 

2.  Determine  the  best  estimate  of  optical  flow. 

3.  Reconstruct  the  3D  geometry  of  the  scene. 

1.  Obtain  the  EPI:  We  first  create  an  EPI  similar  to  Figure  \(b)  in  real  time.  Here  we  describe  how  the  EPI  for  any 
selected  scan  line  is  obtained. 

Figure  2 shows  the  steps  to  obtain  the  EPI  for  a selected  scan  line.  The  desired  scan  line  is  selected  interactively 
using  a program  on  the  host  computer.  Figure  2(a)  shows  the  image  of  an  object  and  the  selected  scan  line  j at  time 
/j.  Figure  2(b)  shows  only  the  scan  line.  This  scan  line  is  spread  to  the  entire  image  as  shown  in  Figure  2(c)  using  the 
image  warping  capability  of  PIPE.  An  image  with  a mask  line  at  the  bottom  is  ANDed  with  the  spread  image  to  shift 
the  scan  line  to  the  bottom  as  shown  in  Figure  2(d).  The  steps  explained  above  are  repeated  for  another  scan  line  at 
the  next  instant  of  time  /2-  Figures  2(e)  through  (h)  show  the  results  at  time  ti,  where  the  image  is  shifted  slightly  to 
the  left  due  to  the  horizontal  motion  of  the  camera.  To  obtain  the  EPI,  the  image  at  time  (Figure  2d)  is  shifted  up 
and  added  to  the  image  at  time  t2  (Figure  2h).  Figure  2(k)  shows  the  result  at  time  /2.  The  process  is  repeated  for  sub- 
sequent scan  lines.  Since  we  are  using  256x242  images,  it  takes  242  cycles  or  8 seconds  to  obtain  the  first  EPI,  but 
subsequent  EPIs  are  formed  in  real-time  by  positioning  the  most  recent  line  at  the  bottom  of  the  EPI  (7=242)  and  shift- 
ing the  previous  scan  lines  up  by  one  row. 

2.  Obtaining  the  best  estimate  of  optical  flow:  This  step  begins  with  the  previously  obtained  EPI.  Figure  3 shows  the 
data  flow  graph  for  this  process.  As  indicated  in  the  left  branch  of  Figure  3,  the  EPI  passes  through  two  Neighborhood 
Operators  (NOPs),  where  it  is  convolved  with  horizontal  and  vertical  edge  operators  GRAD_X  and  GRAD_T  respec- 
tively. The  horizontal  operator  gives  the  spatial  difference  while  the  vertical  operator  gives  the  temporal  difference. 
The  outputs  are  then  directed  to  a Two  Valued  Function  (TVF)  look  up  table  called  SLOPE  which  determines  the 
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(d) 


(h) 


(k) 


scan  line 
at  (j,t2) 


Figure  2:  Steps  to  obtain  the  EPI.  (a)  Image  of  an  object  and  the  selected  scan  line  at  time  /j;  (b)  selected  scan  line  at 
time  /i;  (c)  spread  image  of  (b);  (d)  scan  line  at  time  shifted  to  bottom  of  EPI;  (e),  (f),  (g)  and  (h)  similar  results  at 
time  t2',  (k)  image  (d)  shifted  up  and  added  to  (h). 

slope,  and  hence  the  optical  flow  using  Equation  (11). 

As  mentioned  earlier,  in  order  to  obtain  the  best  estimate  of  optical  flow,  the  zero  crossings  of  the  Laplacian- 
Gaussian  of  the  image  are  used  as  a mask  to  determine  where  optical  flow  is  returned.  The  right  branch  in  Figure  3 
shows  how  the  zero  crossings  are  determined.  The  EPI  is  passed  through  the  difference  of  two  Gaussians  (which  is 
equivalent  to  a Laplacian-Gaussian  operator).  The  output  is  passed  through  a boolean  neighborhood  operator  where  the 
zero  crossings  are  determined.  The  zero  crossings  are  then  ANDed  with  the  optical  flow  image  from  the  TVF  to  obtain 
the  best  estimate  of  the  optical  flow. 

3.  Segmentation  of  optical  flow;  It  has  been  shown  (Equation  (1))  that  optical  flow  is  inversely  proportional  to  the 
depth.  Therefore,  segmenting  the  optical  flow  is  equivalent  to  segmenting  depth.  This  is  performed  by  using  an  interac- 
tive program  from  the  host  computer. 

FOR  each  pixel 

IF  (flow  < threshold ) THEN  intensity  = 0 

ELSE  intensity  = 255 

Consider  two  objects  Oi  and  O2  at  depths  and  d2  from  the  camera,  respectively,  with  (di  > ^2)-  These  give 
rise  to  optical  flow  values  uj  and  U2  (u\  < uf),  respectively.  We  interactively  vary  the  threshold  in  the  program  shown 
above  until  the  flow  due  to  the  object  at  depth  d j disappears  from  the  scene  while  the  flow  due  to  the  object  at  distance 
d2  remains.  We  thus  say  that  we  have  discriminated  between  objects  and  02-  By  physically  measuring  the  values 
di  and  d2,  we  can  calculate  the  range  discriminability  r^  using  the  following  equation  obtained  from  Equation  (1). 

rd{%)  = xlOO  (12) 
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Figure  3;  Data  flow  graph  to  determine  the  slopje. 

To  compute  the  statistics,  we  have  developed  computer  programs  on  the  Sun  computer.  We  obtain  the  EPI  and 
the  zero  crossings  using  the  PIPE.  We  use  the  Sun  for  3-D  reconstruction  of  the  scene  and  subsequent  statistical 
analysis. 


6.  EXPERIMENTS,  RESULTS  AND  DISCUSSION 

A top  view  of  the  setup  for  our  experiments  is  shown  in  Figure  4.  We  use  an  optical  rail  which  has  two  plat- 
forms as  shown  in  the  figure:  one  translates  horizontally  and  the  other  rotates  about  a vertical  axis.  The  camera  is 
mounted  on  the  platforms  as  shown.  The  accuracy  of  the  velocity  of  translation  is  within  0.02%  of  the  selected  velo- 
city. 

We  experimented  with  3x3,  5x5,  and  7x7  edge  operators.  Since  PIPE  performs  only  3x3  neighborhood  opera- 
tions, we  first  used  3x3  edge  operators.  The  experiment  was  repeated  using  5x5  and  7x7  edge  operators.  5x5  and  7x7 
edge  operators  are  approximated  by  products  of  3x3  convolutions.^^ 

Objects  (Oj,  O2)  were  placed  at  different  depths  from  the  camera  (di,  and  the  optical  flow  was  determined. 
Segmentation  was  performed  on  the  optical  flow  and  the  range  differential  was  calculated  The  experiment  was 
repeated  by  decreasing  the  difference  d\-d2  until  further  segmentation  was  not  possible.  The  optical  flow  was  thres- 
holded  to  determine  the  discrim  inability  of  the  method.  The  range  discriminability  calculated  using  Equation  (12)  was 
18%. 
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To  compare  the  different  methods,  we  determine  the  number  of  misclassif ied  points (N^)  for  each  method. 

is  determined  as  follows. 

The  two  objects  0^  and  02  are  placed  as  shown  in  Figure  4.  The  threshold  for  making  the  optical  flow  due  to 
object  Oi  disappear  is  determined.  Knowing  this  value  (threshold),  a three-level  segmentation  is  performed  as  follows. 

FOR  each  pixel 

IF  (flow  < noise  -level ) THEN  intensity  = 0 

IF  (flow  < threshold)  THEN  intensity  = 128 

ELSE  intensity  = 255 

This  program  segments  an  image  such  that  the  image  will  have  an  intensity  value  of  128  for  the  object  in  the 
background  and  255  for  the  one  in  the  foreground.  However,  at  points  where  there  is  no  optical  flow,  the  image  inten- 
sity is  zero.  To  determine  N„^ , a 5x5  window  of  the  image  is  considered.  Since  the  two  objects  in  the  scene  are  far 
apart,  the  image  intensity  in  the  small  window  is  expected  to  be  either  255  or  128  and  zero.  The  total  number  of  non- 
zero points  are  considered  "classified  points".  However,  if  the  intensity  values  in  the  selected  window  are  255,  128, 
and  zero,  then  the  points  with  intensity  value  255  and  those  with  intensity  value  128  are  summed  separately.  The  one 
having  a lower  value  for  the  sum  is  considered  "misclassified".  The  procedure  is  repeated  for  the  entire  image  and  the 
total  number  of  misclassified  points  as  well  as  the  total  number  of  non-zero  points  is  determined.  N„^  is  calculated 
using  the  following  equation. 

^ _ total  number  of  misclassified  points 

total  number  of  points  in  the  image 

I^rmss  was  7.1%  when  the  3x3  operator  was  used.  When  the  5x5  operator  was  used,  reduced  significantly  to 
3.6%.  When  the  7x7  operator  was  used,  N^^  was  2.9%.  The  reduction  in  N„^  was  not  significant  in  the  latter  case, 
because  the  approximate  convolution  had  a small  error.  However,  the  experiment  proved  that  the  results  can  be 
improved  by  using  convolutions  of  higher  order. 

The  following  experiments  to  be  described  involved  using  optical  flow  to  determine  the  depth  from  the  camera. 
First,  K was  determined  as  follows.  If  an  object  is  placed  at  a known  depth  d from  the  camera  and  the  resulting  optical 
flow  u is  determined,  then  K can  be  determined  from  Equation  (1).  Since  it  is  difficult  to  physically  measure  the  exact 
depth  of  a single  object  from  the  focal  point  of  the  camera,  we  used  two  objects  at  depths  di  and  di  from  the  cam- 
era. The  difference  in  the  depths  dx-d2  was  accurately  measured  using  a ruler.  The  optical  flow  values  U]  and  U2  due 
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10  the  two  objects  were  determined  by  manually  measuring  their  corresponding  slopes  in  the  resultant  EPl.  K was 
determined  using  the  following  equation. 

^ ^ (d2-dx)u^U2 
(U1+U2) 

The  subsequent  experiments  were  repeated  with  the  same  camera  settings  and  the  same  velocity,  and  hence  the 
same  ~K.  We  used  two  objects  Oj  and  02  at  depths  d^  and  J2  and  determined  the  depth  from  the  camera  using  3x3, 
5x5,  and  7x7  operators.  We  also  computed  the  mean  and  the  standard  deviations  of  the  results  obtained  from  our 
experiments.  The  results  are  summarized  below. 


Results  Using  Edge  Operators  (Farther  Object) 

Size 

Mean  (mm) 

Standard  Deviation  (mm) 

3x3 

1022 

463 

5x5 

1239 

104 

7x7 

1337 

64 

Results  Using  Edge  Operators  (Nearer  Object) 

Size 

Mean  (mm) 

Standard  Deviation  (mm) 

3x3 

909 

589 

5x5 

975 

102 

7x7 

1065 

45 

Our  results  show  that  better  results  can  be  obtained  by  increasing  the  size  of  the  edge  operators.  The  results  from 
3x3  operators  have  a large  error  in  the  mean,  and  a wide  deviation  from  the  mean.  The  results  can  be  improved  by 
using  5x5  and  7x7  edge  operators. 


7.  CONCLUSION 

Depth  from  motion  can  be  obtained  in  a variety  of  ways,  using  different  techniques.  EPI  analysis  can  be  used  to 
determine  optical  flow  using  edge  operators.  The  algorithm  has  been  implemented  on  PIPE  and  results  have  been 
obtained  in  real  time.  The  accuracy  of  the  method  is  dependent  on  the  size  of  the  edge  operator  used,  but  real  lime 
implementation  on  PIPE  becomes  difficult  as  the  size  of  the  operator  increases.  The  optical  flow  determined  w'ill  have 
high  values  near  the  edge  and  becomes  indeterminate  for  uniform  regions,  with  the  best  estimate  being  at  the  center  of 
the  edge.  Zero  crossings  of  the  Laplacian-Gaussian  of  the  image  have  been  used  to  localize  these  edge  points.  Statisti- 
cal analysis  of  the  results  of  the  edge  operators  confirms  the  fact  that  better  vaues  for  the  mean  and  the  standard  devia- 
tion of  the  depth  can  be  obtained  by  increasing  the  size  of  the  edge  operators. 
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